Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

pkg/utils: speed up LoadBackupMeta (#744) #745

Merged
merged 2 commits into from
Mar 1, 2021
Merged

pkg/utils: speed up LoadBackupMeta (#744) #745

merged 2 commits into from
Mar 1, 2021

Conversation

ti-srebot
Copy link
Contributor

@ti-srebot ti-srebot commented Feb 19, 2021 •
edited
Loading

cherry-pick #744 to release-4.0
You can switch your code base to this Pull Request by using git-extras:

# In br repo:
git pr https://github.com/pingcap/br/pull/745

After apply modifications, you can push your change to this PR via:

git push [email protected]:ti-srebot/br.git pr/745:release-4.0-e901b9eddeb4

What problem does this PR solve?

Speed up LoadBackupMeta.

Cc #705

Manually tested with 39k tables, it improves fast checksum about 53 times.

Benchmark results 
# master branch
[2021/02/19 14:50:58.551 +08:00] [INFO] [collector.go:60] ["Full backup Success summary: total backup ranges: 117738, total success: 117738, total failed: 0, total take(Full backup time): 2m43.027476154s, total take(real time): 23m31.342884193s, total kv: 117738000, total size(MB): 11298.47, avg speed(MB/s): 69.30"] ["backup checksum"=18.410789857s] ["backup fast checksum"=19m20.755603256s] ["backup total regions"=117824] [BackupTS=423025980821995522] [Size=4608796580]

# this branch
[2021/02/19 16:29:03.509 +08:00] [INFO] [collector.go:60] ["Full backup Success summary: total backup ranges: 117738, total success: 117738, total failed: 0, total take(Full backup time): 2m34.65939439s, total take(real time): 4m22.869289236s, total kv: 117738000, total size(MB): 11298.47, avg speed(MB/s): 73.05"] ["backup checksum"=18.999134215s] ["backup fast checksum"=22.223445159s] ["backup total regions"=117824] [Size=4608796580] [BackupTS=423027824506961921]

# master branch
PASS: schema_test.go:218: testSchemaSuite.BenchmarkLoadBackupMeta1024          1        2805279107 ns/op
PASS: schema_test.go:229: testSchemaSuite.BenchmarkLoadBackupMeta10240         1      428872156932 ns/op
PASS: schema_test.go:207: testSchemaSuite.BenchmarkLoadBackupMeta64          200           9807260 ns/op

# this branch
PASS: schema_test.go:218: testSchemaSuite.BenchmarkLoadBackupMeta1024         50          29454271 ns/op
PASS: schema_test.go:229: testSchemaSuite.BenchmarkLoadBackupMeta10240         1        1150945026 ns/op
PASS: schema_test.go:207: testSchemaSuite.BenchmarkLoadBackupMeta64         1000           1662240 ns/op

What is changed and how it works?

Check List

Tests

  • Unit test

Related changes

  • Need to cherry-pick to the release branch

Release Note

  • Improve backup performance when there are many tables.

@ti-srebot ti-srebot mentioned this pull request Feb 19, 2021
Merged
@kennytm
Copy link
Collaborator

kennytm commented Feb 19, 2021

/lgtm

@ti-srebot ti-srebot added the status/LGT1 LGTM1 label Feb 19, 2021
@overvenus overvenus added this to the v4.0.12 milestone Feb 20, 2021
@ti-srebot ti-srebot added status/LGT2 LGTM2 and removed status/LGT1 LGTM1 labels Mar 1, 2021
@overvenus
Copy link
Member

/merge

@ti-srebot
Copy link
Contributor Author

/run-all-tests

@ti-srebot ti-srebot merged commit 22d421a into pingcap:release-4.0 Mar 1, 2021
@overvenus overvenus mentioned this pull request Mar 29, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@overvenus overvenus overvenus approved these changes

@3pointer 3pointer Awaiting requested review from 3pointer

@kennytm kennytm Awaiting requested review from kennytm

@lance6716 lance6716 Awaiting requested review from lance6716

Assignees
No one assigned
Labels
component/backup status/can-merge status/LGT2 LGTM2 type/performance type/4.0-cherry-pick
Projects
None yet
Milestone
v4.0.12
Development

Successfully merging this pull request may close these issues.
3 participants
@ti-srebot @kennytm @overvenus